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Executive Summary 


In response to the increasing importance of technology in education, in 2008 the National 
Assessment Governing Board (NAGB) initiated the development of the National Assessment of 
Educational Progress (NAEP) Technology and Engineering Literacy (TEL) framework to assess 
the technology knowledge and skills of 4th-, 8th-, and 12th-grade students in the United States 
(National Assessment Governing Board 2018, p. 12). Following the development of the 
framework, the National Center for Education Statistics (NCES), within the U.S. Department of 
Education, developed the TEL assessment. The assessment was designed to “capture those 
aspects of the nature, processes, uses, and effects of technology that are particularly important to 
participation in the economic, civic, and social spheres of modern society” (National Assessment 
Governing Board 2018, p. xvi). In 2014, NCES administered the assessment to approximately 
21,500 8th-grade U.S. students. 


Because the effects of technology on education are global and have been building for decades, 
the International Association for the Evaluation of Educational Achievement (IEA) has 
conducted studies on information and communication technologies in education since 1989! and 
began developing the International Computer and Information Literacy Study (ICILS) 
framework in 2010 to assess students’ ability to use computer technology. ICILS reports on the 
computer competency and information literacy skills that students need to participate in the 
digital world. This inaugural administration in 2013 assessed 60,000 8th-grade students across 21 
countries (the United States did not participate). 


In 2018, TEL and ICILS were administered to two separate samples of 8th-grade U.S. students 
of approximately 15,400 and 9,000 students each, respectively. To help understand the TEL and 
ICILS results, NCES conducted a comparison study to answer two research questions: 


e How similar (or different) are the assessment framework targets of TEL and ICILS? 
e How similar (or different) are the characteristics of the TEL and ICILS assessment 
items? 


1TEA conducted the Computers in Education Study (COMPED) in 1989 and again in 1992. The Second Information 
Technology in Education Study (SITES) was conducted in 1998-99, 2001, and 2006. These studies aimed to 
measure the application of computer technology in education by schools and teachers. 
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The results of the comparison study showed that the framework targets and design features 
underlying the scenario-based tasks (SBTs) used in both assessments have many similarities. 
Both assessments have a focus on information and communication technologies, and both are 
digital-based assessments that use interactive features in item design. However, there are 
differences between the two assessments in framework scope and item characteristics. About 
half of the TEL framework targets are beyond the scope of ICILS; TEL evaluates students’ 
knowledge and skills in analyzing general systems, while ICILS limits its evaluation to computer 
systems. ICILS uses some response modes (e.g., flowcharts) that are not used in TEL. Forty-two 
percent of the ICILS items do not allow students to go back to the previous item to change their 
answers, compared with 85 percent of the TEL SBT items. About two-thirds of ICILS items 
assess procedural knowledge, while only about one-third of TEL SBT items do that. 


These similarities and differences need to be considered if students’ performance in the two 
assessments is to be compared. The limitations pertaining to using the study findings to explain 
performance similarities or differences are also discussed at the end of the study. 


Introduction 

In 2018, the National Center for Education Statistics (NCES) administered two assessments—the 
National Assessment of Educational Progress (NAEP) Technology and Engineering Literacy 
(TEL) assessment and the International Computer and Information Literacy Study (ICILS)—to 
two separate nationally representative samples of 8th-grade students in the United States. TEL 
was administered in January—March to 15,400 students, while ICILS was administered in 
March—May to 9,000 students. Because of some similarities in content and item formats, NCES 
commissioned a study to explore the similarities and differences between the two assessment 
frameworks as well as the characteristics of the assessment items. 


About TEL 

The TEL assessment was administered first in 2014 and then again in 2018 to 8th-grade students 
in both public and private schools in the United States. The samples of schools and students were 
chosen to produce representative data for schools and students across the nation.” 


In the TEL framework, technology is defined as "any modification of the natural world done to 
fulfill human needs or desires," and engineering is defined as "a systematic and often iterative 
approach to designing objects, processes, and systems to meet human needs and wants" 
(National Assessment Governing Board 2018, p. xvi). The framework defines technological and 
engineering literacy as "the capacity to use, understand, and evaluate technology as well as to 
understand technological principles and strategies needed to develop solutions and achieve 
goals" (p. 5). The TEL framework measures three content areas: Technology and Society, 
Design and Systems, and Information and Communication Technology. 


Technology and Society refers to how technology affects society and the natural world? and the 
ethical questions stemming from those effects. Design and Systems includes “the nature of 
technology, the engineering design process by which technologies are developed, and basic 
principles of dealing with everyday technologies, including maintenance and troubleshooting” 
(National Assessment Governing Board 2018, p. xvii). Information and Communication 
Technology covers “computers and software learning tools, networking systems and protocols, 
hand-held digital devices, and other technologies for accessing, creating, and communicating 
information and for facilitating creative expression” (p. xvii). These content areas cut across 
three practices*: Understanding Technological Principles, Developing Solutions and Achieving 
Goals, and Communicating and Collaborating. Students are presented with problems from TEL 
content areas that require them to apply TEL practices. 


2 For more information about NAEP TEL, visit https://nces.ed.gov/nationsreportcard/tel/ 

3 The TEL framework defines the “natural world” as “plants, animals, water, and other organisms and elements that 
exist without contributions from humans” (National Assessment Governing Board 2018, p. 129). 

4 “Practices” as used in the TEL framework refer to particular ways of thinking and reasoning that students are 
expected to be able to apply to solve problems (National Assessment Governing Board 2018, p. xviii). 
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Using scenario-based tasks (SBTs) in addition to traditional discrete items, TEL aims to measure 
students’ ability to apply technology and engineering skills to real-life situations. The SBTs 
employ real-world scenarios to engage students in technology and engineering problems 
spanning multiple, interdependent items. The information from interdependent items is enriched 
by the addition of discrete, standalone items. 


Students taking the TEL assessment use NCES-provided laptop computers and are asked to 
complete two 30-minute blocks. Each block is composed of either one SBT or a combination of 
a short SBT and discrete items. The assessment also surveys students about their experiences 
with technology (both in and outside of school) and school administrators about technology 
resources and practices. 


About ICILS 

ICILS, first administered in 2013 with 21 countries or education systems and again in 2018 with 
14 countries or education systems including the United States, is an international, computer- 
based assessment of 8th-grade students sponsored by the International Association for the 
Evaluation of Educational Achievement (IEA). ICILS defines its international target populations 
based on the number of years of formal schooling that students have received. Specifically, 
ICILS includes students who are in their 8th year of education. In many countries, students reach 
their 8th year of schooling when they are 13.5 years old or above. ICILS also administers 
questionnaires to 8th-grade teachers (randomly sampled), principals, and the ICT coordinators of 
sampled schools.° 


ICILS 2013 explored international differences in students’ Computer and Information Literacy 
(CIL), and ICILS 2018 was expanded to assess students’ Computational Thinking (CT) as an 
optional component. The United States administered both the CIL and CT components to 
students in 2018.° 


The ICILS 2018 Assessment Framework (figure 1) defines Computer and Information Literacy 
(CIL) as "an individual's ability to use computers to investigate, create, and communicate in 
order to participate effectively at home, at school, in the workplace, and in society" (Fraillon et 
al. 2019, p. 1). The framework defines Computational Thinking (CT) as “an individual’s ability 
to recognize aspects of real-world problems which are appropriate for computational formulation 
and to evaluate and develop algorithmic solutions to those problems so that the solutions could 
be operationalized with a computer” (p. 1). The measurement of CT was offered as an option in 
the 2018 ICILS administration. 


5 For more information about ICILS, visit https://nces.ed.gov/surveys/icils/. 
® Unlike TEL, ICILS does not have a single scale—CIL and CT are measured and reported separately. 
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Figure 1. International Computer and Information Literacy Study (ICILS) 2018 
Framework 
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SOURCE: International Association for the Evaluation of Educational Achievement (IEA), International Computer and 
Information Literacy Study (ICILS), 2018. 


The ICILS cognitive assessment consists of testing “modules.” Like TEL blocks, students are 
given 30 minutes to complete each ICILS module. Students taking the ICILS assessment have 
120 minutes to complete four modules. Each module contains multiple items contextualized in a 
single scenario and resembles a scenario-based task in TEL. The items at the beginning of a 
module, each called “a short task” in ICILS, are less complex than the items that follow and 
typically take about a minute to finish. The last item in a module, called “a large task,” is usually 
more complex and takes more time to complete. For example, students may be asked to create a 
poster or presentation. 


ICILS surveys students, teachers, school administrators and ICT coordinators to understand the 
use of technology in teaching and learning, both in and out of school. ICILS also includes a 
national context questionnaire—posed to national ICILS research coordinators, who draw upon 
national experts to answer the questions—which collects data on educational policies and 
guidelines related to the use of technology in schools. 


Comparing TEL and ICILS 

Administered in the spring of 2018 to 8th-grade U.S. students, both assessments were presented 
on computer equipment provided by NCES. The results of TEL 2018 were released in April 
2019 and the results of ICILS were released in November 2019. Table 1 below provides a 
summary of the key elements of the two assessments administered in 2018. 


Table 1. Similarities and differences between key elements of TEL and ICILS 


Assessment aspects TE (OIES 
Assessment focus Technology and engineering Computer and Information 
literacy Literacy; and Computational 
Thinking 
Assessment sponsor U.S. Department of International Association for the 
Education Evaluation of Educational 
Achievement 
U.S. assessment National Center for National Center for Education 
administrator Education Statistics Statistics 
Assessment scope Domestic International 
Assessment time 60 mins for cognitive items 120 mins for cognitive items 
Survey questionnaire Student (15 mins) | Student (30 mins) 
School (25 mins) School (20 mins) 
| Teacher (30mins) 
Population 8th-grade students 8th-grade students 
U.S. sample size Schools (600) Schools (300) 
Students (15,400) Students (9,000) 
| Teachers (4,500) 
Sample drawn January—February 2017 May 2017 
Administration window — January—March 2018 _March—May 2018 
Assessment equipment NCES-supplied laptops NCES-supplied tablets 
Accommodations for Available Not available 
students in assessment 
delivery platform | 
Reporting window April 2019 November 2019 


SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 
(NAEP) Technology and Engineering (TEL) Assessment, 2018; and International Association for the Evaluation of Educational 
Achievement (IEA), International Computer and Information Literacy Study (ICILS), 2018. 


Purpose of the study 
Given the similarities between the 2018 TEL and ICILS, two pressing research questions were 
raised: 


e How similar (or different) are the assessment framework targets of TEL and ICILS? 


e How similar (or different) are the characteristics of the TEL and ICILS assessment 
items? 


Answering these questions will enable NCES and U.S. stakeholders to better understand the 
nature and comparability of the 2018 TEL and ICILS assessments. This information is necessary 
for NCES to address questions raised by policymakers and educators, produce special reports for 
certain stakeholders, and improve the quality and resource efficiency of the two assessments in 
future cycles. 


Benefits of the framework comparison 

Comparing the TEL and ICILS frameworks has two main benefits. First, similarities between the 
frameworks suggest areas where TEL data can provide assessment measures and achievement 
data on students’ grasp of the technology and engineering principles described in the ICILS 
framework, and vice versa. Second, differences between the two frameworks suggest areas 
where the TEL and ICILS assessments can provide unique measures of students’ knowledge and 
abilities. Identifying the comparable and incomparable framework targets enables a more 
nuanced analysis of students’ performance on the TEL and ICILS assessments. 


Beyond the differences in framework scope, the two assessments also employ different design 
principles for items and test delivery systems. Therefore, the American Institutes for Research 
(AIR) also conducted an analysis of the two assessments’ item pools in this study in addition to 
the framework target comparison. 


Benefits of the item format comparison 

Whereas the TEL and ICILS frameworks represent the intentions of each respective assessment, 
the items embody the implementation of each framework. Differences in item formats can affect 
the actual participating student sample or have effects on student performance. For example, 
TEL items are presented on a digital platform that offers accessibility tools such as 
magnification, text-to-speech read-aloud, and high-contrast presentation mode.’ ICILS does not 
have these tools, so participating students who need accommodations can be excluded from the 
assessment per participating schools’ discretion. 


7 For more information about student accommodations and inclusion, visit 
https://www.nationsreportcard.gov/tel/about/samples-inclusion-participation/. 
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Study Methodology 


This study consists of two parts: an assessment framework comparison of TEL and ICILS, and 
an analysis of the item design features of the two assessments. Both parts of the study were 
concurrently analyzed in three stages. 


In Stage 1, TEL and ICILS framework targets were matched based on the scope of the concepts 
or topics measured in the targets, and assessment items were coded based upon their features. 
Multiple team members individually matched the framework targets and coded the item 
characteristics, and then convened to adjudicate any differences. 


In Stage 2, the team convened a panel meeting of experts—familiar with technology literacy, 
interactive computer technology, national and international assessments, and the cognitive fields 
underlying these assessments—from October 18 to 22, 2017, to provide input on the comparison 
between TEL and ICILS. A panel meeting occurred with four researchers familiar with TEL, 
ICILS, or similar technology literacy frameworks.® 


The panel members were asked to provide input on (1) the alignment of the two frameworks 
upon which the assessments are built, regrouping the targets when needed and providing their 
individual rating of the groupings, (2) methods to capture the characteristics of the items 
employed by the two assessments, and (3) plans for reporting and sharing the comparison study 
results as context for the assessment results.° 


In Stage 3, the AIR research team averaged experts’ individual ratings of the assessment target 
groups. The item characteristic codes were updated based upon the coding categories modified 
by the panel. 


Aligning the TEL and ICILS frameworks 

In Stage 1, the TEL framework targets were matched to ICILS framework targets. 
Comparatively, most of the ICILS framework targets cover broader areas than most of the TEL 
framework targets, allowing one or more TEL framework targets to fit under a single ICILS 
target. 


In Stage 2, the expert panel evaluated these preliminary groupings on a 5-point scale, ranging 
from 0 to 4.'° Table 2 shows how each score was defined. 


8 One panel member dropped out a day before the panel meeting due to illness, so the final panel contained four 
members. 

° Participating panelists’ biographical statements can be found in appendix A. The agenda for the meeting can be 
found in appendix B. 

1° For more information about the methodology, visit 
https://nces.ed.gov/nationsreportcard/subject/science/pdf/ngss_naep_ highlights report.pdf. 
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Table 2. Framework groupings: Ratings and definitions 


Rating Definition 
4 Overlapping with same focus 
3 Overlapping with somewhat similar focus 
2 _ Overlapping, but slightly different focus 
1 Overlapping, but very different focus 
0 _ No overlap 


The panelists first individually rated the proposed framework groupings and then discussed the 
compiled ratings and disagreements. In the initial discussion, panel members sometimes 
suggested alternate groupings from those proposed, sometimes agreeing to add or remove 
framework targets. Finally, the panelists provided a second rating for each proposed grouping 
and provided their rationale before advancing to the next framework grouping. 


The panelists were asked to consider the following questions when assigning similarity ratings 
for each framework grouping: 


= What is the primary focus/emphasis of the concepts measured in this group? 

= What is the breadth of content covered? 

= What is the depth of content covered? 

" How advanced is the content? 

« Are specific portions of the possible content in one framework included/excluded in the 
other? 

= What types of items/tasks could be developed to assess the content in the group? 


Analysis of item characteristics 
The analysis of item characteristics began with a review of the TEL and ICILS item pools. The 
resulting TEL-ICILS Item Classification Guide (see appendix C) yielded 11 categories: 


1. Framework Coverage: How each assessment organizes the competencies (content area 
and/or practices) covered by each item 

2. Context: The scenario/topic being used to engage respondents. The specific contexts are 
varied, so for TEL-ICILS comparison purposes, contexts are categorized into 1 of 10 
possible buckets. 

3. Related Subject: An 8th-grade subject that informs/teaches the content and/or 
competencies being assessed. 

4. Design Elements: Media elements used in the item and stimulus information. 

5. Item Response Mode: The method(s) that respondents must use to provide responses. 
Additional notes are added to the subcategory column when necessary. 

6. Scrolling: Any scrolling used in an item. 


7. Navigation Linearity: The freedom of movement the respondent has within the task (for 
scenario-based tasks only).'! 

8. Interactive Features: A classification of whether the item and stimulus information 
contain interactive features (e.g., a model that respondents can use to run trials in order to 
answer items, or elements that students can add or remove). 

9. Scoring Level: The number of scoring levels contained in the item’s scoring rubric. 

10. Scoring Method: A classification of items measured by whether they are scored by 
humans or machines. 

11. Item Knowledge Type: The type of knowledge presented in the item (including the 
stimulus or stem of the item). 


These categories do not contain assessment-level features. '* For example, TEL implements 
systemwide universal-design accommodations, such as text-to-speech, background color 
contrasts, and zoom capability in the assessment delivery platform, while ICILS does not contain 
any comparable tools. 


Using the 11 item-level characteristics contained in the preliminary TEL-ICILS Item 
Classification Guide, all TEL and ICILS items were coded and summary statistics were produced 
for each assessment. The expert panel reviewed the preliminary item classification guide and 
summary statistics and recommended minor changes affecting four existing coding categories: 
Framework Coverage, Related Subject, Item Response Mode, and Design Elements. The 

changes were as follows: 


" For the Framework Coverage category, the panelists suggested coding all ICILS items 
onto TEL practices to provide one more dimension for comparison. 

" For the Related Subject category, the panelists’ only question was the difference between 
Health and Science curricula for 8th-graders and whether “health” might be a better fit 
for some of the biology-related items currently coded as “science.” 

" For the Jtem Response Mode category, the panelists had concerns that “artistic” might be 
more accurately titled “design elements,” and they suggested relabeling it as “creative.” 
Also, as there is only one instance of a drop-down menu item, the panelists advised that it 
would be better placed under the multiple-choice single-selection category, since students 
are still selecting from provided options. 

= Panelists’ comments regarding Design Elements included changing “none” to “text only” 
and renaming the category “Item Presentation Elements” (originally named “Item 
Presentation Media” and ultimately named “Design Elements”). The panel also 
recommended breaking up the entry for simulated computer applications and changing 
the entry title to “User Interface Navigation.” 


'l When students interact with TEL discrete items, they can answer the items in any order they wish because there is 
no scenario requiring a beginning, middle, or end. This feature of TEL discrete items is assessment wide and as such 
is not calculated at an item level. 

'2 Although these assessment-spanning differences did not need to be coded item by item, it is important to note that 
they almost certainly affect construct measurement and could lead to differences in student performance. Thus, 
assessment-spanning differences should be taken into consideration when comparing the results of the two 
assessments. 
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The panel did not advise adding or removing a new category. In Stage 3, the panels’ input was 
incorporated into an updated TEL-ICILS Item Classification Guide. The final coding of the TEL 
and ICILS items to each category shows the panel’s input. 


Results 


The results of the framework and item format comparisons show that although there is some 
overlap between the targets in the TEL and ICILS frameworks, about half of the TEL framework 
targets are beyond the scope of ICILS targets. Analysis of the item pools shows that the two 
digital-based assessments use many innovative features in item design, as both include similar 
response modes, media elements, and interactive features. However, the different item features 
could lead to very different student experiences. The two assessments may favor different 
students based on their previous skills or knowledge. The findings from the comparison of the 
two frameworks and their assessment items are discussed next. 


Framework similarities and differences 

Among 47 targets in the TEL framework and 12 targets in the ICILS framework, 26 TEL targets 
are grouped with ICILS targets because of overlapping content, forming a total of 12 groupings. 
The average rating of panel members is 3 or 4 for 10 out of the 12 groupings, indicating similar 
content and focus. Of the remaining groupings, 2 were given a rating of 1. Lastly, the remaining 
21 TEL targets are not grouped with any ICILS targets because of a lack of clear overlap. In 
general, the panel observed that TEL’s focus on “technology” is broader than ICILS’s focus on 
“computers.” Despite this broader focus, the average TEL individual framework target focuses 
on a more specific aspect of knowledge or skills than an individual ICILS target. As a result, the 
panelists often remarked that the clusters of TEL framework targets align better with ICILS 
targets than do the individual components of those clusters. 


Based on the experts’ ratings and comments, the grouping and alignment of the TEL and ICILS 
assessment targets were updated. Table 3 shows a summary of the finalized TEL and ICILS 
framework target groupings and ratings (an expanded table of the TEL and ICILS groupings can 
be found in appendix D). 
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Table 3. Framework groupings and expert panel ratings 


Grouping # —_ICILS target TEL target Rating! | 
1 CTI. D.8.11, D.8.12, D.8.15, 1.8.9,4 D.8.13 3 
2 CTi2 D.8.8,D.8.9,D.8.17 . 1 
3 CT1.3 D.8.9, L8.5, 1.8.8, 1.8.13 4 
4 CT2.1 D.8.6, D.8.7, D.8.8, D.8.14 4 
5 CT2.2 D.8.8 1 
6 CILLA 1.8.13 3 
ee emer 18.4, 18.6 3 
8  CIL22 1.8.5, 1.8.13, T.8.8° << 
9 | CIL3.1 1.8.3, L8.13, T.8.11 4 
10 C32 1.8.3, L8.13, T.8.11 3 
i CIL4.1 1.8.3, 18.12, 1.8.13, T.8.10 4 
12 CIL4.2 _T.8.12, 7.8.14, T.8.15, 18.10, 18.11 3 


d Ratings range from 0 to 4. 


> Based on the ICILS framework naming convention, CT1.1 refers to the 1st aspect in the Ist strand of the Computational 
Thinking (CT) framework. 


3 Based on the TEL framework naming convention, D.8.11 refers to the 11th assessment target for 8th grade in the Design and 
System area. 


41.8.9 refers to the 9th assessment target for 8th grade in the Information Communication Technology area. 
> CIL1.1 refers to the 1st aspect in the Ist strand of the CIL framework. 
® T.8.8 refers to the 8th assessment target for 8th grade in the Technology and Society area. 


The highest rated groupings feature overlapping content and focus, while also making similar 
cognitive demands of students. For example, CIL3.1, which targets transformation of 
information to make it audience-friendly/ready for consumption, was rated as having substantial 
overlap with the TEL clusters of 1.8.3, 1.8.13, and T.8.11, which ask students to manipulate 
information for communication purposes. The panel determined that all these targets were 
making similar demands (manipulation/transforming data) and for the same purpose (effective 
communication). 


The groupings that received ratings of “3” contain framework targets with some overlapping 
content and some overlap in the aptitude required of students. As an example, the CIL3.2 and 
1.8.3, 1.8.13, and T.8.11 grouping was rated a “3” because the targets make similar demands on 
students to manipulate data for communication, but the ICILS target focuses on data specifically 
in the computer context whereas the TEL targets do not expressly address computer use. 
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Groupings that have some overlap but received lower ratings often have some content similarity, 
but the specific focus and/or student requirements differ. For example, CT1.2 deals with 
formulating and analyzing problems, whereas D.8.8, D.8.9, and D.8.17 address specific aspects 
of problem solving, such as constructing models and troubleshooting processes. The panel 
determined that the topic was similar, but the focus and demands differed. Some of the ICILS 
problem-solving processes did not align with TEL; while the TEL processes could be construed 
as subcomponents of the ICILS statement, they were not specified in the ICILS statement. TEL 
D.8.8 and ICILS CT2.2 also received an average rating of only 1 because the ICILS target 
focuses specifically on computer coding while the TEL target is about general design process. 


Many of the TEL framework targets rated as having zero comparability with any ICILS 
framework targets and thus were not in any groups (see table 4). Most of the Technology and 
Society targets from TEL were deemed to have no overlap with ICILS, which does not have a 
focus on impact/sustainability. TEL also has collaboration, feedback, and citation targets, which 
fall outside of the scope of ICILS. Panelists found some TEL targets to be very specific (e.g., 
redesigning tools in D.8.5), making them hard to put into any TEL cluster that more broadly 
aligns with ICILS. 


Table 4. Ungrouped TEL framework targets 


TEL content area Framework target 


D.8.1, D.8.2, D.8.3, D.8.4, D.8.5, 


Design and Systems D.8.10, D.8.16, D.8.18, D.8.19 


Information and Communication 


Technology 18.1, 1.8.2, 1.8.7 


Technology and Society T87T89T8 
Item characteristics: Similarities and differences 

The 2018 TEL has a total of 184 items—107 across the SBTs and 77 as discrete items. Unlike 
the scenario-based items, discrete items bear close resemblance to traditional stand-alone items 
found in paper-based assessments. Given this difference, information about discrete items is 
presented separately from information about SBT items. The 2018 ICILS has a total of 113 items 
across all modules (i.e., SBTs). Tables 5 and 6 show the item distributions of the two 
assessments. 
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Table 5. TEL items, by content area 


TEL content area SBT Discrete 
Design and Systems Ti 23 23 
Information and Communication 
44 32 
Technology | 
_ Technology and Society =| 40 a) 
| Total 107 items 77 items 


Table 6. ICILS items, by component 


ICILS component SBT | 
Computational Thinking 18 
| Computer and Information Literacy | 95 | 
| Total 113 items | 


In addition, both TEL and ICILS items were coded in terms of the subjects to which they are 
related, TEL practices, item context, item response mode, scoring level, scoring method, item 
design elements, scrolling, navigation linearity, interactive features, and item knowledge type. 
Each of these characteristics is examined below. 


“Related subject” refers to an 8th-grade subject that informs/teaches the content and/or 
competencies being assessed in an item. The results showed that about half of the ICILS items 
tap into computer science or technology classes, while 41 percent of TEL SBT items tap into 
science and 34 percent into social studies (table 7). 


Table 7. Percentage of TEL and ICILS items, by related subject 
Related subject ICILS TELSBI TEL Discrete 


Math 2% 2% 1% 
ELA | 12% 2% 11% 
Science | 22% AM% 43% 
Social Studies 16% 34% 43% 
Computer Science 49% a 16% 3% 
“Art | 0% | TH 0% 
‘Total 100% (113 items) 100% (107 items) 100% (77 items) 
NOTE: Detail may not sum to totals because of rounding. 


In addition to content area targets, assessments of practices !* are required in the TEL framework 
but are not part of the ICILS framework. To make comparisons, ICILS items were also coded 
into TEL practices. As shown in table 8, about half of the ICILS items assess the practice of 
developing solutions and achieving goals (DSAG), while about two-thirds of TEL SBT items 
assess this practice. The distribution of practices assessed across the ICILS items is similar to the 
distribution of practices assessed across TEL discrete items. 


Table 8. Percentage of TEL and ICILS items, by TEL practice 


TEL practice ICILS TEL SBT (NEED Merete 
Communicating and 17% 17% 27% 
Collaborating | 

Developing Solutions 7 : ‘ 
and Achieving Goals oie ns 10a 
Understanding A ‘ 
Technological Principles ahs we A 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


All items from both assessments were also coded for context (i.e., the story or topic used to 
engage students) (see table 9). In the development of the TEL items, the proposed context was 
reviewed to ensure that it would not be unfamiliar to any student groups. In ICILS, the proposed 
context was reviewed by participating countries for appropriateness. The coding results show 
that about half of the ICILS items are presented in the context of recreation and afterschool 
activities, followed by environment and sustainability and health and biosciences. TEL items 
cover a larger variety of contexts. In addition to the contexts employed by ICILS, TEL presents 
items in the context of government, industries, engineering, business, and school. 


'3 TEL Practices is a TEL framework term that refers to specific approaches to thinking and reasoning to solve 
problems. 
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Table 9. Percentage of TEL and ICILS items, by context 


Context (IES TEL SBI TEL Discrete 
Government 0% 10% 9% 
Recreation and r ‘ji ‘ 
afterschool activities a ane ee 
Industries 0% 0% 3% 
Agriculture 8% 11% 1% 
Environment and j 4 - 
sustainability ance as ae 
ere 14% 0% 9% 
biological sciences 

Engineering 0% 21% 17% 
Business 0% 16% 17% 
Transportation 8% 0% 8% 
School 0% 15% 10% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


Response mode refers to the format in which the assessment requires students to provide 
responses, such as the selection of answers from a set of options, drag-and-drop answers, and 
select-and-explain. Across the two assessments, the constructed-response format is used 
frequently: 29 percent in ICILS, 29 percent in TEL discrete, and 21 percent in TEL SBT (see 
table 10). In addition, 26 percent of ICILS items use creative work or presentation, while 27 
percent of TEL SBT use the select-and-explain response format. These response modes give 
students great flexibility to provide rationales for their answers, which reflects the nature of 
technology use—in most cases, multiple solutions are possible. TEL discrete items employ more 
traditional response modes to assess what students know; as such, it is not surprising to see that 
39 percent of the items are single-selection multiple-choice questions. 
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Table 10. Percentage of TEL and ICILS items, by item response mode 


Item response mode ICILS TEL SBT TEL Discrete 
Single-selection multiple-choice 10% 18% 39% 
Multiple-selection multiple-choice 0% 9% 6% 
Drag-and-drop 12% 10% 6% 
Select-and-explain 4% 27% 9% 
Selection cluster 2% 8% 10% 
Constructed response | 29% 21% 29% 
Action 13% 0% 0% 
Flowchart 2% 0% 0% 
Creative work or presentation 26% 1% 0% 
Other 1% 6% 0% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


Table 11 contains a summary of scoring levels used in the two assessments. Two-score level 
items mean students either get the answer correct or incorrect. Three scoring levels mean 
students can get full credit, partial credit, or zero credit. Items with more scoring levels are 
usually more complex than items with fewer scoring levels. For TEL discrete items, a majority 
(60 percent) of the items have only two scoring levels. For TEL SBT items, about half of the 
items have three scoring levels and only 4 percent have 5 or 6 scoring levels. In ICILS, 42 
percent of the items have two or three scoring levels, respectively, and about 10 percent have 
eight or more scoring levels. 


Table 11. Percentage of TEL and ICILS items, by scoring level 


Scoring level ICILS TEL SBT TEL Discrete 
2 42% 36% 60% 
3 42% 47% 40% 
4 6% 13% 0% 
5 0% 2% 0% 
6 0% 2% 0% 
8 3% 0% 0% 

10 3% 0% 0% 

12 4% 0% 0% 

Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 
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Regarding scoring methods, TEL and ICILS SBT items are similar—about half of both are 
scored by machines and half by human scorers (table 12). A higher percentage (62 percent) of 
TEL discrete items were scored by machines than were TEL SBT items. 


Table 12. Percentage of TEL and ICILS items, by scoring method 


NYofoyunatcap nate rere! ICILS TEL SBT TEL Discrete 
Human scored 50% | 49% 38% 
Machine scored 50% 51% 62% 
Total 100% (113 items) | 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


In terms of item design features, aside from the systemwide universal design for accommodation 
purposes in TEL, 47 percent of TEL SBT items involved either data tables, avatars, or audio files 
(table 13). In contrast, 50 percent of ICILS items included either a user interface navigation or a 
web page. Both assessments use simulations (4 to 6 percent of the items) that required students 
to solve problems by adjusting relevant variables. 


Table 13. Percentage of TEL and ICILS items, by design element 


Design element ICILS SBT TEL SBT TEL Discrete 
Pictures 22% 19% 39% 
Graphs 5% 8% 6% 
Data tables | 0% 14% | 17% 
Simulations | 4% 6% 6% 
ae 20% % 0% 
Web pages 30% 9% 5% 
Avatar 6% 20% | 0% 
E-mails 1% 6% 0% 
Video 9% 4% 4% 
Audio 0% 13% 0% 
Text only 1% 0% 0% 
Other 2% 2% 24% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 
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As a design principle, TEL generally avoids scrolling in items to make the item presentation as 
user-friendly as possible. As a result, only 5 percent of TEL SBT items require scrolling to see 
all content, whereas 50 percent of ICILS items require scrolling (table 14). Multiple studies have 
shown that students perform better when they do not need to scroll to view all the pertinent item 
information (Pommerich 2004 and 2007; Texas Education Agency 2008). 


Table 14. Percentage of TEL and ICILS items, by scrolling in item 


Scrolling in item ICILS EAE SET TEL Discrete 
No 50% 95% 78% 
Yes 50% | 5% 22% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


Table 15 shows the summary of the navigation linearity of the items in the ICILS and TEL 
SBTs.'* In both assessments, correct answers need to be presented to “level” incorrect students 
with correct students, allowing all of them to move forward. When the leveling occurs, students 
are not allowed to go back to the previous item to change their answers. As the table shows, 42 
percent of the ICILS items do not allow students to go back, compared with 85 percent of the 
items in TEL SBTs. This comparison indicates TEL provides more leveling than ICILS. 


Table 15. Percentage of TEL and ICILS items, by navigation linearity 
Navigation linearity ICILS TEL SBT 


Forward only 42% 85% 
Forward and backward 58% 15% 
Total 100% (113 items) | 100% (107 items) 


NOTE: Navigation linearity is not applicable to TEL discrete items, since they are stand-alone 
items. Detail may not sum to totals because of rounding. 


Although many different design elements are used in both assessments, not all of them are 
interactive (e.g., an image). The percentage of items with interactive features is summarized in 
table 16. About 67 percent of TEL SBT items had interactive features, compared to 50 percent of 
ICILS items. As expected, TEL discrete items had the lowest percentage of interactive features. 


Table 16. Percentage of TEL and ICILS items, by interactive feature 


DilicrecKeahom ier litlas ICILS TEL SBT TEL Discrete 
No 50% 33% 95% 
Yes | 50% 67% 5% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


“4 Navigation linearity is not applicable to TEL discrete items, since they are stand-alone items. 
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The knowledge involved in the items was also compared across the two assessments. Table 17 
shows that two-thirds of TEL SBT items focus on declarative knowledge, asking students to state 
facts, while two-thirds of ICILS items focus on procedural knowledge, asking students to go 
through the steps or procedures needed to solve problems. See appendix C, which shows the item 
classification guide, for more information about declarative and procedural item classifications. 


Table 17. Percentage of TEL and ICILS items, by item knowledge type 
Item knowledge type ICILS TEL SBT TEL Discrete 


Declarative | 33% | 67% 82% 
Procedural 67% | 33% 18% 
Total 100% (113 items) 100% (107 items) 100% (77 items) 


NOTE: Detail may not sum to totals because of rounding. 


To summarize, the items in the two assessments share many similar features—using item 
response modes that allow flexibility and multiple solutions; tapping into knowledge in science 
and social studies subjects; employing similar media elements; using contexts of recreation, 
afterschool activities, environment, and sustainability; and having a similar proportion of items 
scored by machines and humans. However, there are also many differences in the focus, design, 
and presentation of items. Compared with ICILS, TEL uses a more diverse context to present 
items, focuses more on interactive features in SBTs, requires more data interpretation skills, 
provides more leveling, and avoids scrolling in items. ICILS has more complex items, assesses 
more computer interface skills, and focuses more on procedural knowledge and skills than TEL 
does. See appendix E for a sample item classification using a TEL item, appendix F for a sample 
ICILS item, and appendix G for a summary table of the TEL and ICILS item characteristic 
categories. 
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Limitations of the Comparison Study 

There are limitations to the findings from this framework and item characteristics comparison 
study. The limitations mainly pertain to using these findings to explain similarities or differences 
in student outcomes between the two assessments. The limitations to framework-level 
comparisons stem from the fact that framework measurement objectives are targets—they are the 
constructs that are intended to be assessed. However, the resulting assessments built upon the 
frameworks may differ in focus and scope; this difference would not be captured by framework 
comparison studies, which compare measurement targets, not what is actually measured. 
Furthermore, among the TEL and ICILS framework targets that were found to overlap, the 
instruments used to measure them differ. For instance, TEL and ICILS items that measure 
comparable framework targets may differ in context, depth of measurement (scoring level), and 
response mode. A TEL item may solicit a constructed response, whereas an ICILS item 
measuring a comparable framework target may employ a multiple-choice response format. A 
student may perform differently on the constructed-response item than on a multiple-choice item 
given the different stimulus or clues they can get from the item wording. Moreover, the 
constructed-response item may have a partial-scoring level whereas the multiple-choice item 
may only allow for correct and incorrect scores. It is possible that a student who scores 
“incorrect” on the multiple-choice format would be able to achieve at least a partial score on a 
constructed-response item. 


In addition to the item-level difference between TEL and ICILS, there are assessment-level 
differences that could lead to variations in performance. TEL provides systemwide 
accommodations—such as text-to-speech, background color contrasts, and zoom capability— 
that are built into the assessment delivery platform. These universal-design accommodations 
(e.g., a digital read-aloud tool) are provided to all U.S. students who participate in NAEP digital 
assessments, regardless of special needs. This differs from ICILS, which does not have such 
accommodations built into the test delivery platform but allows extra time and intermittent 
reading aloud if needed. Some students with special needs who needs accommodations may have 
been excluded from participating in ICILS.'° The different approach to accommodations 
produced a different student experience and may have resulted in a slightly different student 
sample. 


'S The U.S. student exclusion rate met international requirements (under 5%). 
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Additional, assessment-spanning differences include the computer equipment used to deliver the 
digital assessments. In the United States, students took the TEL assessment on laptops with a 
mouse attached, whereas the ICILS assessment was administered on tablets!* paired with 
attachable keyboards and mice. The equipment for both tests administered in the United States 
was supplied by NCES, ensuring standardization in the administrations of the assessments. 
Conversely, there was no standardization of the equipment used in the international ICILS 
samples beyond minimum equipment requirements (screen size, keyboard, mouse, etc.). It is thus 
possible that differences in devices may have driven some of the differences in student 
performance. For example, students may have typed differently on the thin, chiclet-style 
keyboards attached to the ICILS tablets than on the large keyboards used in the TEL laptops. 


Conclusions 

The results of this study show that about half of the TEL framework targets fall within the scope 
of the ICILS framework, and the other half fall outside of it. The most similar content areas 
measured in the two assessments are information and communication technologies. However, 
there are differences. TEL does not focus on computational thinking and ICILS does. TEL 
evaluates students’ knowledge and skills about analyzing general systems, while ICILS limits its 
evaluation to computer systems. TEL includes any technology and its impact on society as one of 
three framework content areas, whereas ICILS focuses primarily on ethical issues related to 
computer technology and internet use. 


There is significant overlap in the item formats and characteristics employed by the two 
assessments. However, there are many different features and design considerations that may 
confound direct comparison of student performance. As such, the comparability of student 
performance could be enhanced with further analysis of scoring and standards-setting practices. 
In addition, comparisons of subpopulations of participating students would yield more 
comparable results (for example, a comparison of TEL students who eschewed or lightly used 
the available accommodation features to ICILS students who did not have those 
accommodations). 


The findings of this study can be the basis for future efforts to incorporate assessment results into 
the comparison analysis to help researchers and practitioners better understand the differences 
and similarities of students’ performances in TEL and ICILS. 


‘6 Although the ICILS assessment was delivered on touch-screen tablets in the United States, the touch-screen 
interface was disabled. As with traditional laptops, the only way students could interact with the devices was 
through keyboards and mice. 
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Appendixes 
Appendix A. List of Expert Panelists 


Esther Care 
Senior Fellow, Center for Universal Education, Brookings Institution 


Esther Care works to promote effective assessment practices that inform both policy and 
classroom practice. She has worked primarily in the Asia Pacific with a focus on providing evidence- 
based advice to ministries of education implementing or planning to implement major education reform. 
Dr. Care’s work is characterized by consideration of the interactions between assessment, curriculum, and 
pedagogy. With the increasing focus globally on the need to promote generic skills, such as problem 
solving, critical thinking and collaboration, she has worked with ministries of education to identify how to 
incorporate these skills into the educational process. 


Dr. Care is a director of the Assessment Curriculum and Technology Research Centre (funded by 
the Australian Government’s Department of Foreign Affairs and Trade, in the Philippines), which 
conducts large- and small-scale research in the Philippines to inform that country's major K-12 education 
reform. This work emphasizes the dependencies across curriculum, assessment, and pedagogy and is 
designed to provide quick feedback to the government on the roll-out of the reforms. 


Dr. Care has had long-term working relationships with Australian education providers in early 
literacy development and working with teachers to improve skills in data interpretation and use to drive 
student learning. Resulting from her research as chief investigator on several Australian Research Council 
grants, she has published in journals and books spanning vocational and educational psychology to 
education assessment and policy, most recently with an emphasis on the assessment of 21st century skills. 


Leigh Ann DeLyser 
Director of Education and Research, CSNYC 


Leigh Ann DeLyser is the director of education and research at the NYC Foundation for 
Computer Science Education (CSNYC). In this role, Dr. DeLyser is working to expand computer science 
to all schools in the New York City public school system. CSNYC is the private partner in the $80 
million initiative requiring every school to offer one unit of computer science to every student in public 
schools. She is a coauthor of the Running on Empty report, a 50-state analysis of computer science 
standards. Prior to obtaining her doctorate in computer science and cognitive psychology from Carnegie 
Mellon University, Dr. DeLyser was a high school computer science and math teacher and a two-term 
member of the board of directors of the Computer Science Teachers Association. She also helped start the 
Academies for Software Engineering in New York City as a proof of concept that all students could learn 
computer science. 
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Dr. DeLyser is a lifelong advocate of computer science education. At CSNYC, she oversees 
research efforts and advising and implementing programs that align with the organization’s strategic 
goals. She is also a co-chair of the CSforAll Consortium, a national network of computer science 
education content providers, school districts, education associations, and researchers devoted to the 
mission of CSforAll. Dr. DeLyser earned her Ph.D. from Carnegie Mellon University in Computer 
Science and Cognitive Psychology, with a focus on computer science education. 


Sara Dexter 


Associate Professor 
Department of Leadership, Foundations & Policy 
Curry School of Education, University of Virginia 


Sara Dexter is an associate professor of education with expertise in technology leadership. Dr. 
Dexter’s experience in administration at the school district level brings a unique perspective to the 
Administration and Supervision faculty concerning the systemic implementation and integration of 
technology at the district, school, and classroom levels. Her first-hand experience in designing and 
implementing professional development on the instructional use of technology for a public school district 
in Minnesota is reflected in her passion for bringing awareness, understanding, cohesion, and research to 
the exciting nature of technology leadership. 


Dr. Dexter’s primary research interests lie in exploring the relationship between educational 
leadership, technology leadership, and resource management. She has an extensive background in the 
development of case-based learning environments for both aspiring teachers and school administrators. A 
current federally funded grant supports the development, implementation, and research of web-based, 
interactive cases for the development of pre-service administrators’ decisionmaking skills. Her research 
incorporates decisionmaking skills at multiple levels of leadership regarding technology implementation. 


Through her courses, she guides school leaders in strategically designing systems in which 
students’ needs might be best served by the leadership structures used for implementation of technology 
in schools. An integral part of her instruction includes facilitating the process of honing school leaders’ 
skills to discern how to tackle technology implementation through the collaborative utilization of human 
resources and the use of available technology within their schools. Based on her innovative research in 
technology leadership, she was the 2009 recipient of the University Council for Educational 
Administration’s Jack A. Culbertson Award, presented annually to an outstanding junior professor of 
educational administration, in recognition of her contributions to the field. 
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Don Knezek 
Senior Global Education Consultant 


Don Knezek, former CEO of the International Society for Technology in Education (ISTE), is 
recognized internationally for his leadership in transforming learning and teaching through effective and 
innovative uses of technology. He has led innovation at all levels of education, from classrooms and 
school districts to state, national, and international projects. Recent leadership efforts include directing a 
national center for teacher preparation; consulting with ministries of education and affiliated groups from 
all over the world on digital age standards; and partnering with UNESCO on its ICT Competency 
Framework for Teachers and related credentialing programs for 21st century teaching. Dr. Knezek holds a 
bachelor’s degree cum laude from Dartmouth College, a master’s degree from the University of Hawaii, 
and a Ph.D. from the University of Texas. He is a tireless advocate for universal education. 


26 


Appendix B. Expert Panel Meeting Agenda 


AGENDA 


TEL-ICILS Comparison Study: Expert Panel 
October 18-20, 2017 | Washington, DC 


American Institutes for Research (AIR) 
1025 Thomas Jefferson Street NW 
Room 3131 


Contact Number: (202) 403-6568 


Wednesday, October 18, 2017 


1:00 pm 
1:30 pm 
2:00 pm 


3:15 pm 
3:30 pm 
4:00 pm 
5:00 pm 


Registration 

Welcome and Introductions — everyone 

Introduction of TEL and ICILS — Taslima Rahman, William Ward, Lydia 
Malley from NCES 

Break 

Overview of the Study, Review Materials and Procedures — Yan Wang 
Practice Framework Review 

Adjourn 


Thursday, October 19, 2017 


9:00 am 
9:15 am 
11:00 am 
11:15 am 
12:30 pm 
1:30 pm 
2:45 pm 
3:00 pm 
4:30 pm 
5:00 pm 


Summary of Prior Day’s Discussion 

Comparison of ICILS and TEL Frameworks — Panel/AIR 

Break 

Comparison of ICILS and TEL Frameworks (cont.) — Panel/AIR 
Lunch 

Comparison of ICILS and TEL Frameworks (cont.) — Panel/AIR 
Break 

Comparison of ICILS and TEL Frameworks (cont.) — Panel/AIR 
Debrief 

Adjourn 


Friday, October 20, 2017 


8:30 am 
9:00 am 
10:00 am 
11:00 am 
11:30 am 
12:30 pm 
1:30 pm 
2:30 pm 
2:45 pm 
3:00 pm 
4:00 pm 
4:30 pm 


Summary of Prior Day’s Discussion 

ICILS and TEL Items and Their Characteristics (preliminary results) — Yan Wang 
Discussion of the Coding Categories/Scheme — Panel/AIR 

Break (hotel check-out) 

Discussion of the Coding Categories/Scheme (cont.) — Panel/AIR 
Lunch 

Panel Suggestions on Additional/Future Analyses — Panel 

Break 

Current Plans for Reporting Study Results — Taslima Rahman 
Panel Suggestions on Reporting and Dissemination — Panel 
Debrief 

Adjourn 
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Appendix C. TEL-ICILS Item Classification Guide 


A. Framework Coverage: How each assessment organizes the competencies covered by each 


item. 


a. 


b. 


d. 


Content Area/Dimension: Primary-level framework content found in TEL and ICILS 
Content Subarea/Strand: Secondary-level framework content found in TEL and 
ICILS 

Assessment Target/Aspect: Tertiary-level framework content found in TEL and 
ICILS 

Practice: Skills target found in TEL 


B. Context: The scenario/topic being used to engage respondents. The specific contexts are 
varied, so for TEL-ICILS comparison purposes, contexts are categorized into | of 10 
possible buckets. 


1. 


OAS oe wt 


2 


Government (voting, legislation, etc.) 

Recreation and afterschool activities (hobbies, outdoor activities, etc.) 
Industries (industrial fishing, computer industry, etc.) 

Agriculture (farming, farming science, etc.) 

Environment and Sustainability (alternative energy, recycling, etc.) 
Health and Biosciences (biology, body functions, etc.) 

Engineering (machine processes, systems, etc. ) 

Business (cost or revenue related to a business) 

Transportation (car travel, bus system, etc.) 


10. School (class assignment, school infrastructure, etc.) 
C. Related Subject: An 8th-grade subject that informs/teaches the content and/or competencies 
being assessed. 


Ne 


Math 

English Language Arts 
Science 

Social Studies 


Computer Science 
Art 
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D. Item Design Elements: Media elements used in the item and stimulus information. 
Pictures 
Graphs (including flowcharts) 
Data tables 
Simulations 
User interface navigation (computer system and application navigation) 
Web pages 
Avatar 
E-mails 
9. Video 
10. Audio 
11. Text only 
E. Response Mode: The method(s) that respondents must use to provide responses. The 
response modes are arranged as follows. Additional notes are added to the subcategory 
column when necessary. 
1. Single-Selection Multiple-Choice: A multiple-choice question with one answer. 
2. Multiple-Selection Multiple-Choice: Multiple-choice question with multiple correct 
answers possible. Must select all correct answers for full credit. 
3. Drag-and-drop: Item where answer is created by dragging responses into a grid or 
table. 
4. Select-and-Explain: Item where answer is a multiple-choice question followed by a 
constructed-response explanation. 
5. Selection Cluster: Series of binary items, scored together as one item. 
6. Constructed Response: Item where response is a typed answer only. Also, items 
where a hyperlink or other item must be copied/pasted. 
7. Action: an action (e.g., click on a link, select a file) is captured and scored. 
8. Flowchart: Item where response is a multi-selection from a chart or graph. Selections 
are individual clicks. 
9. Creative work or presentation: Items that are determined by design ability. Examples 
include appropriate picture placement, text/background contrast, and editing a photo. 
10. Other: Items that did not fit into any categories above (e.g., sorting, data entry, drop- 
down menu, drag an object). 
F. Scrolling: Any scrolling used in an item gets marked as a ‘yes.’ 
1. Yes 
2. No 
G. Navigation Linearity: (For scenario-based tasks only), the freedom of movement the 
respondent has within the task. 
1. Forward only: Test does not allow you to return to previously answered items. 
2. Forward & backward: Test allows for going back to prior items or advance to any or 
some future items. 


CON NST ee aS oS 
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H. Interactive Features: A classification of whether the item and stimulus information contain 
interactive features (e.g., a model that respondents can use to run trials in order to answer 
items). Note that videos, while re-playable, are not considered “interactive;” voiceover 
buttons for avatars are consider interactive. 

1. Yes 
2. No 

I. Scoring level: The number of scoring levels the item’s scoring rubric contains. 

1. Score level 2 

Score level 3 

Score level 4 

Score level 5 

Score level 6 

Score level 8 

Score level 10 

8. Score level 12 

J. Scoring Method: A classification of items measured by whether they are scored by humans 
or machines. 

1. Human: Items that don’t have fixed answers and need human grading based on 
scoring criteria, such as a constructed response. 

2. Machine: Items where credited answers or keys are fixed and predetermined, such as 
single-selection multiple-choice items. 

K. Item Knowledge Type: The type of knowledge presented in item (including the stimulus or 
stem of the item). 

1. Declarative: The knowledge required to answer the item is factual and informational, 
regardless of what students are asked to do with it. Examples include opinions, data 
tables, background information of a debate. Students can be asked to take notes, 
summarize content, or make a choice. 

2. Procedural: The knowledge required to answer the item is about steps or sequence for 
how things progress or how to perform certain activities, regardless of what students 
are asked to do with it. Examples include steps to test a design, a video to show stages 
of a plant’s growth, a flowchart that shows a food web. Students can be asked to take 
notes, summarize content, or make a choice. 


STD OB Be 
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Appendix D. Groupings of ICILS and TEL Framework Targets, Ratings, and Rationale 


ICILS TEL 
framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 


Knowing about and Technological systems are designed to achieve goals. Somewhat different focus, but 
understanding They incorporate various processes that transform inputs aptitude required is similar. 
computer systems into outputs. They all use energy in some form. These 
refers to a person’s processes may include feedback and control. 

ability to identify and 

describe the properties |D.8.12 Technological systems can interact with one another to 
perform more complicated functions and tasks than any 
individual system can do by itself. 

interaction of the 

components within a Construct and use a moderately complicated system, 
system. given a goal for the system and a collection of parts, 
including those that may or may not be useful in the 
system. 


Use a digital model of a system to conduct a simulation. 
Explain how changes in the model result in different 
outcomes. 


Examine a product or process through reverse 
engineering by taking it apart step by step to identify its 
systems, subsystems, and components, describing their 
interactions, and tracing the flow of energy through the 
system. 
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ICILS TEL 
framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 


Formulating problems 
entails the 
decomposition of a 
problem into smaller 
manageable parts and 
specifying and 
systematizing the 
characteristics of the 
task so that a 
computational solution |D.8.17 
can be developed 
(possibly with the aid 
of a computer). 
Analyzing consists of 
making connections 
between the properties 
of, and solutions to, 
previously experienced 
problems and new 
problems to establish a 
conceptual framework 
to underpin the process 
of breaking downa 
large problem into a 
set of smaller, more 
manageable parts. 


Carry out a design process to solve a moderately 
difficult problem by identifying criteria and constraints, 
determining how they will affect the solution, 
researching and generating ideas, and using trade-offs to 
choose between alternative solutions. 


Construct and test a model and gather data to see if it 
meets the requirements of a problem. 


Diagnose a problem in a technological device using a 


logical process of troubleshooting. Develop and test 
various ideas for fixing it. 
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ICILS requires problem-solving 
processes not specified in TEL. Both 
require an understanding of problem 
solving, but the demands made of 
students are different. 


ICILS 
framework 
target # 


ICILS aspect 
description 


In order to make 
effective judgments 
about problem solving 
within systems, it is 
necessary to collect 
and make sense of 
data from the system. 
The process of 
collecting and 
representing data 
effectively is 
underpinned by 
knowledge and 
understanding of the 
characteristics of the 
data and of the 
mechanisms available 
to collect, organize, 
and represent the data 
for analysis. 


TEL 
framework 
target # 


Similarity 
TEL statement description rating 


Construct and test a model and gather data to see if it 
meets the requirements of a problem. 


Select and use appropriate digital and network tools and 
media resources to collect, organize, analyze, and display 
supporting data to answer questions and test hypotheses. 


Use digital tools to gather and display data in order to 
test hypotheses of moderate complexity in various 
subject areas. Draw and report conclusions consistent 
with observations. 


Use appropriate digital tools to accomplish a variety of 
tasks, including gathering, analyzing, and presenting 
information as well as creating text, visualizations, and 
models and communicating with others. 
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Rationale 


The TEL statements together form the 
components of the ICILS target 
(collecting, analyzing, and displaying 
data). 


ICILS 
framework 
target # 


TEL 
ICILS aspect framework 
description target # 


Planning solutions 
refers to the process of 
establishing the 
parameters of a 
system, including the 
needs of users and 
desired outcomes, with 
a view to designing 
and implementing the 
key features of a 
solution. Evaluating 
solutions refers to the 
ability to make critical 
judgments about the 
quality of 
computational artifacts 
(such as algorithms, 
code, programs, user 
interface designs or 
systems) against 
criteria based on a 
given model of 
standards and 
efficiency. 


TEL statement description 


Engineering design is a systematic, creative, and 
iterative process for meeting human needs and wants. It 
includes stating the challenge, generating ideas, 
choosing the best solution, making and testing models 
and prototypes, and redesigning. Often there are several 
possible solutions. 


Requirements for a design are made up of the criteria for 
success and the constraints, or limits, which may include 
time, money, and materials. Designing often involves 
making trade-offs between competing requirements and 
desired design features. 


Carry out a design process to solve a moderately 
difficult problem by identifying criteria and constraints, 
determining how they will affect the solution, 
researching and generating ideas, and using trade-offs to 
choose between alternative solutions. 


Measure and compare the production efficiency of two 


machines, a simple machine and a complex machine, 
designed to accomplish the same goal. 
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Similarity 
rating 


Rationale 


This cluster of TEL statements rates 
higher than the individual targets 
because of their specificity in 
comparison to ICILS’s relative 
broadness. Topic and focus are very 
similar. 


ICILS 
framework 
target # 


ICILS aspect 
description 


TEL 
framework 
target # 


his aspect focuses on |D.8.8 


he logical reasoning 
hat underpins the 
development of 
algorithms and code to 
Solve problems. It can 
involve developing or 
implementing an 
algorithm as well as 
automating the 
algorithm, typically 
sing computer code. 
Creating a design in 
his domain refers to 
he intersection 
between users and the 
system. This may relate 
o development of the 
ser interface elements 
in a program or to the 
design of functional 
specifications or 
requirements about 
lhow a program or 
system should interact 
ith its users. 


Similarity 
TEL statement description rating Rationale 


Carry out a design process to solve a moderately difficult |1 ICILS is specifically about code and 
problem by identifying criteria and constraints, algorithms, while TEL is about design 
determining how they will affect the solution, iprocess. While there is a little overlap, 
researching and generating ideas, and using trade-offs to he focuses are different. 

choose between alternative solutions. 
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ICILS TEL 
framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 


owing about and Use appropriate digital tools to accomplish a variety of While the aptitude required to meet 
nderstanding computer tasks, including gathering, analyzing, and presenting he requirements is similar in both 
se refers to a person’s information as well as creating text, visualizations, and argets, the focus is different. ICILS 
declarative and models and communicating with others. focuses on the data whereas TEL 
focuses on the tools. 


procedural knowledge 
of the generic 
characteristics and 
functions of computers. 
his aspect focuses on 
he basic technical 
owledge and skills 
hat underpin our use of 
computers in order to 
ork with information. 
Accessing and 8. Increases in the quantity of information available through The TEL statements together have a 
evaluating information electronic means and the ease by which knowledge can very similar focus, and slightly 
refers to the be published have heightened the need to check sources different demand of students (ICILS 
investigative processes for possible distortion, exaggeration, or emphasizes being able to do something; 
hat enable a person to Imisrepresentation. TEL emphasizes knowing how to do 
find, retrieve, and make something). 
judgments about the 8. Search media and digital resources on a community or 
relevance, integrity, world issue and identify specific examples of distortion, 
land usefulness of exaggeration, or misrepresentation of information. 


computer-based 
information. 
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ICILS 
framework 
target # 


ICILS aspect 
description 


TEL 
framework 
target # 


Managing information [1.8.5 


refers to the capacity 
of individuals to work 
ith computer-based 
information. The 
process includes the 
lability to adopt and 


adapt information 
classification and 
organization schemes 
in order to arrange and 
store information so 
hat it can be used or 
reused efficiently. 


TEL statement description 


Select and use appropriate digital and network tools and 
media resources to collect, organize, analyze, and display 
supporting data to answer questions and test hypotheses. 


Use appropriate digital tools to accomplish a variety of 
tasks, including gathering, analyzing, and presenting 
information as well as creating text, visualizations, and 
models and communicating with others. 


Information technologies are developing rapidly so that 


the amount of data that can be stored and made widely 
accessible is growing at a faster rate each year. 
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Similarity 
rating 


Rationale 


Some of the TEL focus does not 
overlap with ICILS (e.g., the "creating" 
in 1.8.13), but the focus and student 

equirements regarding data 
management are similar. 


! 


ICILS TEL 


framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 
Communicate information and ideas effectively using a The focus on maximizing information 
information refers to a variety of media, genres, and formats for multiple ransformation is very similar. The 


iperson’s ability to use purposes and a variety of audiences. purpose for transforming data is similar 
computers to change as well (effective communication) 

how information is 8. Use appropriate digital tools to accomplish a variety of although ICILS does not have the focus 
presented so that it is tasks, including gathering, analyzing, and presenting lon collaboration that TEL does (e.g., 
clearer for specific information as well as creating text, visualizations, and T.8.11). 

audiences and models and communicating with others. 


ipurposes. This process 
Use appropriate information and communication 


technologies to collaborate with others on the creation 
and modification of a knowledge product that can be 


ypically involves 
sing the formatting, 


graphics, and 
multimedia potential accessed and used by other people. 


of computers to 
enhance the 
communicative effect 
or efficacy of 
information. 
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ICILS 
framework 
target # 


ICILS aspect 
description 


Creating information 
refers to a person’s 
ability to use 
computers to design 
and generate 
information products 
for specified purposes 
and audiences. 

These original 
products may be 
entirely new or may 
build upon a given set 
of information to 
generate new 
understandings. 


TEL 
framework 
target # 


TEL statement description 


Communicate information and ideas effectively using a 
variety of media, genres, and formats for multiple 
purposes and a variety of audiences. 


Use appropriate digital tools to accomplish a variety of 
tasks, including gathering, analyzing, and presenting 
information as well as creating text, visualizations, and 
models and communicating with others. 


Use appropriate information and communication 
technologies to collaborate with others on the creation 
and modification of a knowledge product that can be 
accessed and used by other people. 
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Similarity 
rating 


Rationale 


Significant overlap in the manipulation 
of data for communication. Slightly 
different focus between TEL's use of 
data tools and ICILS’s broader focus 
on data/information in the specific 
context of computers. 


ICILS 
framework 
target # 


ICILS aspect 
description 


Sharing information 
refers to a person’s 
understanding of how 
computers are and can 
be used as well as his 
or her ability to use 
computers to 
communicate and 
exchange information 
with others. Sharing 
information focuses 
on a person’s 
knowledge and 
understanding of a 
range of computer- 
based communication 
platforms, such as e- 
mail, wikis, blogs, 
instant messaging, 
sharing media, and 
social networking 
websites. 


TEL 
framework 
target # 


TEL statement description 


Communicate information and ideas effectively using a 
variety of media, genres, and formats for multiple 
purposes and a variety of audiences. 


Certain digital tools are appropriate for gathering, 
organizing, analyzing, and presenting information, 
while other kinds of tools are appropriate for creating 
text, visualizations, and models and for 
communicating with others. 


Use appropriate digital tools to accomplish a variety of 


tasks, including gathering, analyzing, and presenting 
information as well as creating text, visualizations, and 
models and communicating with others. 


The large range of personal and professional information 


technologies and communication devices allows for 
remote collaboration and rapid sharing of ideas 
unrestricted by geographic location. 
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Similarity 
rating 


Rationale 


While TEL focuses more on tools, 
ICILS focuses more on information. 
However, all of the statements in this 
grouping target the same domain: 
information sharing. 


ICILS 
framework 
target # 


ICILS aspect 
description 


Using information 
responsibly and safely 
refers to a person’s 
understanding of the 
legal and ethical issues 
of computer-based 
communication from 
the perspectives of 
both the publisher and 
the consumer. 


TEL 
framework 
target # 


TEL statement description 


Technology by itself is neither good nor bad, but its use 
may affect others; therefore, decisions about products, 
processes, and systems must take possible consequences 
into account. 


Explain that it is important for citizens to reduce the 
negative impacts and increase the positive impacts of 
their technologies on people in another area or on future 
generations. 


Explain why it is unethical to infect or damage other 
people’s computers with viruses or "hack" into other 
computer systems to gather or change information. 


Style guides provide detailed examples for how to give 
appropriate credit to others when incorporating their 
ideas, text, or images in one’s own work. 


Identify or provide examples of fair use practices that 
apply appropriate citation of sources when using 
information from books or digital resources. 


Science is the systematic investigation of the natural 
world. Technology is any modification of the 
environment to satisfy people’s needs and wants. 
Engineering is the process of creating or modifying 
technologies and is constrained by physical laws and 
cultural norms and economic resources. 


41 


Similarity 
rating 


Rationale 


TEL focuses on knowing about the 
ethical and appropriate use of 
"technology," whereas ICILS focuses 
on appropriately using "information." 
Knowing is not the same as doing, and 
although the focus is slightly different, 
there is some overlap. 


ICILS TEL a 
framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 


Technology advances through the processes of innovation 
and invention. Sometimes a technology developed for one 
purpose is adapted to serve other purposes. 


Tools have been improved over time to do more 
difficult tasks and to do simple tasks more efficiently, 
accurately, or safely. Tools further the reach of hands, 
voice, memory, and the five human senses. 


Simulate tests of various materials to determine which 

would be best to use for a given application. 

Redesign an existing tool to make it easier to accomplish er FT LLLLt—SE 
a task 


Communicate the results of a design process and 
articulate the reasoning behind design decisions by 
using verbal and visual means. Identify the benefits of 
a design as well as the possible unintended 
consequences. 


Many different kinds of products must undergo regular 
maintenance, including lubrication and replacement of 
parts before they fail so as to ensure proper functioning. 


Modify a moderately complicated system so that it is less 
likely to fail. Predict the extent to which these 
modifications will affect the productivity of the system. 


Trace the life cycle of a repairable product from 
inception to disposal or recycling in order to determine 
the product’s environmental impact. 
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ICILS TEL 
framework ICILS aspect framework Similarity 
target # description target # TEL statement description rating Rationale 


Collaboration can take many forms. Pairs or teams of 
people can work together in the same space or at a 
distance, at the same time or at different times, and on 
creative projects or on technical tasks. Different 
communications technologies are used to support these 
different forms of collaboration. 


Provide feedback to a (virtual) collaborator on a 

product or presentation, taking into account the other 
person’s goals and using constructive, rather than 
negative, criticism. 

Use digital tools to identify a global issue and investigate 
possible solutions. Select and present the most promising 
sustainable solution. 


Economic, political, social, and cultural aspects of 
society drive improvements in technological products, 
processes, and systems. 


Technology interacts with society, sometimes bringing 
about changes in a society’s economy, politics, and 
culture and often leading to the creation of new needs 
and wants. 

Describe and analyze positive and negative impacts on 
society from the introduction of a new or improved 
technology, including both expected and unanticipated 
effects. 


Compare the impacts of a given technology on different 
societies, noting factors that may make a technology 
appropriate and sustainable in one society but not in 
another. 
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ICILS 
framework ICILS aspect 
target # description 


TEL 
framework 
target # 


TEL statement description 


Some technological decisions involve trade-offs between 
environmental and economic needs, while others have 
positive effects for both the economy and environment. 


Resources such as oceans, fresh water, and air, which are 
essential for life and shared by everyone, are protected 
by regulating technologies in such areas as 
transportation, energy, and waste disposal. 


Compare the environmental effects of two alternative 
technologies devised to solve the same problem or 
accomplish the same goal and justify which choice is 
best, taking into account environmental impacts as well 
as other relevant factors. 


Similarity 
rating 


Information technologies make it possible to analyze and|0 


interpret data, including text, images, and sound, in ways 
that are not possible with human senses alone. These uses 
may result in positive or negative impacts. 


People who live in different parts of the world have 
different technological choices and opportunities because 
of such factors as differences in economic resources, 
location, and cultural values. 


Rationale 


SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2018 Technology and Engineering (TEL) 
Assessment; and International Association for the Evaluation of Educational Achievement (IEA), 2018 International Computer and Information Literacy Study (ICILS), previously 


unpublished tabulations (May 2020). 
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Appendix E. Sample TEL Item and Item Classification 


A sample released TEL item"’ is used here to illustrate item classification categories. 


= iiapeemmene A. Framework Coverage 
Accnum Pilot Block Year Grade Subject Area Item Type Key 


VHO007781 J2TXTM19 2014 8 TEL DS Selection Cluster (see scoring guide) 


Classification Scoring Rubric Pilot Data 


Assessment Area 


Competency 


Framework Target |D.8.8°S f rocess to solve a moderately difficult problem by identifying criteria and constraints, determining how they will 
ct the solution, researching and generating ideas, and using trade-offs to choose between alternative solutions 

Code DS-D03 
Work Product Final design 

= Observable Feature Number of requirements satisfied 

$ 4 

S Upper Category All requirements are satisfied 

vc 

> Performance 

w 
Lower Category None or relatively few requirements are satisfied 
Performance 


Rationale Advanced students can address the requirements in the final design 


E. Item Type: Multiple Selection Multiple Choice B. Context: Transportation 


2 eS 


LIMIT u ging 
45 y a ae lee a C. Related Subject: Science 


rs to perform the following. 
© Change the car's speed limit to 25 mph by 
using the Speed Limit slider 
* Select 14 feet of car lane width by using the 
Car Lane Width slider 


G. Navigation: 
Linear forward only 


When you are finished, click Submit. 


SUBMIT 


H. Interactive Features: Yes 


'7 National Assessment of Educational Progress (NAEP), 2014 Technology and Engineering Literacy (TEL) 
Assessment. Retrieved from https://www.nationsreportcard.gov/tel_2014/#tasks/bikelanes. 
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F. Scrolling: No 
D. Item Presentation Elements: Simulations 


Use the two sliders on the left to see ho 


the car speed and car lane width affect the 
of a road. 


UNSAFE Use the sliders to perform the following. 


Car Lane Width slider 


When you are finished, (lick Submit. 


14 13 12 11 JcarLane Width +——s——+ 


SUBIVIII 


K. Item Knowledge Type: Procedural 
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|. Score Levels: 3 


Correct selections are as follows: 


Car lane width set to 14 ft 
Sued Hedi wattaS¥iook J. Scoring Method 


Yes (the configuration is safe for cyclists) 
either of the two pairs of settings will result in an unsafe cycle lane: 11f & 45 mph or 12 
ft and 45 mph. 
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Appendix F. Sample ICILS Item 


Drone Simulator 


Pattern 


Direction 


Run Simulation 


Starting position (Row, Column) 


The drone must photograph every pumpkin. 
Which flight path (pattern and direction) and starting position (row, column) will result Click iQ] to see the task details again. 
in the drone photographing every pumpkin? 

Use the drone simulator to help answer the question. 


Click BBarnen you are ready to continue. 


Pattern Direction Starting position 
O Grid | O Left | Row Column 
O Spiral © Right L_| 
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Appendix G. Summary Table of Item Comparisons 


Table G-1. Percentage of TEL and ICILS items, by item characteristic 


ronem@tbe-latouriale ONES TEL SBT TEL Discrete 
Related subject 
Math 2% 2% 1% 
ELA 12% 2% 11% 
Science 22% 41% 43% 
Social Studies 16% 34% 43% 
Computer Science 49% 16% 3% 
Art 0% 7% 0% 
p TEL practice 
Communicating and Collaborating 27% 17% LUE 
oo Solutions and Achieving 47% 66% 48% 
eee Technological 27% 17% 25% 
<= S = &86=6=6—hChCttCS 
Government 0% 10% 9% 
Recreation and afterschool activities 50% 21% 13% 
Industries 0% 0% 3% 
Agriculture 8% 11% 1% 
Environment and sustainability 20% 6% 13% 
Health and biosciences 14% 0% 9% 
Engineering 0% 21% 17% 
Business 0% 16% 17% 
Transportation 8% 0% 8% 
School 0% 15% 10% 


Item Characteristic 


Item response mode 


TEI SBT 


TEL Discrete 


Human scored 


Single-selection multiple-choice 10% 18% 39% 
Multiple-selection multiple-choice 0% 9% 6% 
Drag-and-drop 12% 10% 6% 
Select-and-explain 4% 27% 9% 
Selection cluster 2% 8% 10% 
Constructed response 29% 21% 29% 
Action 13% 0% 0% 
Flowchart 2% 0% 0% 
Creative work or presentation 26% 1% 0% 
Other 1% 6% 0% 
2 42% 36% 60% 
3 42% A7T% 40% 
4 6% 13% 0% 
5 0% 2% 0% 
6 0% 2% 0% 
8 3% 0% 0% 
10 3% 0% 0% 
12 4% 0% 0% 


50% 


49% 


| Scoringmethhd method 


38% 


Machine scored 


50% 


51% 


62% 
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inne @etbe-latouliale ICILS TEL SBT TEL Discrete 
Design element 
Pictures 22% 19% 39% 
Graphs 5% 8% 6% 
Data tables 0% 14% 17% 
Simulations 4% 6% 6% 
User interface navigation 20% 0% 0% 
Web pages 30% 9% 5% 
Avatar 6% 20% 0% 
E-mails 1% 6% 0% 
Video 9% 4% 4% 
Audio 0% 13% 0% 
Text only 1% 0% 0% 
Other 2% 2% 24% 


| Scrollinginitem = ss in item 


NOTE: Detail may not sum to totals because of rounding. 


No 50% 95% 78% 
Yes 50% 5% 22% 
Forward only 42% 85% - 
Forward and backward 58% 15% + 
No 50% 33% 95% 
Yes 50% 67% 5% 
Declarative 33% 67% 82% 
Procedural 67% 33% 18% 
Total 113 items 107 items 77 items 
+ Not applicable. 


The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and 
reporting data related to education in the United States and other nations. It fulfills a congressional mandate to 
collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; 
conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state 
and local education agencies in improving their statistical systems; and review and report on education activities in 
foreign countries. 


NCES activities are designed to address high-priority education data needs; provide consistent, reliable, complete, 
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